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TELEOPERATOR SYSTE M AND METHOD WITH TELEPRESENCE^ 

FTFLD OF THE INVENTION 

This invention relates generally to teleoperator 
method and apparatus, and particularly to those which 
include means for providing the operator of remote 
5 apparatus with the same sense as working directly with 
his hands at the worksite. 

^ACXGP.OUNr) OF THE tntveNTION 

Teleooerating, which is well known, includes the 
human performance of tasks at a remote location using 
manipulators. Telepresense includes providing the 
teleooerator with the same feedback and control that he 
would' have were he actually at the worksite carrying ou, 
the operation with his own hands. Telepresence operation 
generally includes use of a stationary visual display, 
particularly a stereographic visual display of the remote 
workspace. Stereoscopic television systems are well 
known' as shown, for example, in U.S. Patent Numbers 
c 562,^63 and 4,533,117 and in U.K. Patent Application 

<^A G3 2,040,134. 

20 Remote manipulators employing stereoscopic TV 

viewing together with force feedback also are well known 

cv,„,,^ ^o- e y?-ni e in an article entitled, 
as shown, ^Oi exe.;..~-e, j." 

••Controlling Remote Manipulators Through Kinesthetic 
Coupling, « Bejczy et ai, Computers in Mechanical 
25 Engineering, July 1933, p?s . 4S-6C, and in an article 
Entitled, "Stereo Advantage fcr a ?eg-In-Hcle TaskUsing 
a Force-Feedback Manipulator" by E.K. Spain, SPIr. Vol. 
1256 Stereoscopic Displays and Applications, 1990, p?s . 
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244-254. in the Bejczy et ai article, force-torque 
feedback is disclosed. Also, in U.S. Patent N'u^oer 
3,921,445, a manipulator which includes force, torque ana 
sli? sensors of a type which may be employed with the 
present invention is shown. 

Even though the operator of prior art manipulators 
is provided with a stationary three-dimensional image of 
the workspace, and manual controllers for control of the 
manipulators are provided with feedback, the operator is 
not provided with a sense of actually being present at 
the worksite. The present invention is directed to a 
viewing arrangement for use in a remote manipulation 
system which substantially adds to the operator's sense 
of presence at the remote manipulator site. 
qTTMMARY AND OBJECTS OF TH - tnvtn'TION 

An object of this invention is the provision of an 
improved telecperator system and method which include an 
imoroved viewing system to enhance the operator's sense 
of" presence at remote manipulators controlled by the 
operator from a remote location. 

An object of this invention is the provision of 
an improved telecperator system and method of the above- 
mentioned type wherein an image of manipulator end 
effectors for viewing by the operator are sensed by the 
operator as comprising an integral part of hand- 
controllers used by the operator to control the end 
effectors, thereby giving the operator a strong sense or 
presence at the worksite. 

An object of this invention is the provision of an 
improved telecperator system and method of the above- 
mentioned tvoe which is well adapted for use in a wice 
variety of applications including military, industrial, 
biomedical, and the like. 

The present invention includes manipulators located 
at a worksite and which are controlled by hand-opera Led 



15 



20 



30 



means at a renote operator control station. . End 
effectors at the manipulators are used for manipulating 
objects located in a workspace at the worksite, and 
fcrce-tcrcue feedback is employed for transmitting back 
5 to the operator mechanical resistance encountered by the 
end effectors. Stereographic visual display neans 
provide the operator with an image of the workspace.' In 
accordance with the present invention, the image is 
located adjacent the hand-operated means so that the 
10 operator locks in the direction of the hand-operated 
means for viewing the image adjacent the hand-operated 
means. Either a real or virtual image of the workspace 
may be provided adjacent the hand-operated means. 
Display means for display of a real image may be located 
adjacent the hand-operated means for direct viewing of 
the real image by the operator. For display of a virtual 
image of the workspace, a mirror is located between the 
operator's eyes and the hand-operated means. In this 
case, display means provide a real image which is 
inverted from top to bottom, which inverted image is 
viewed via the mirror, which mirror inverts the image and 
provides the operator with a virtual image of the 
workspace, which appears to be located adjacent the hand- 
ooerated means. By locating the image of the workspace 
25 adjacent the hand-operated means the operator is provided 
with a sense that the end effectors and hand-operated 
means are substantially integral despite the fact the end 
effectors are located at the worksite and the hand- 
operated means are located at the remote operator's 
station. A stereophonic sound system may be included to 
□rovide the operator with stereophonic sound from the 
worksite. Video camera means are provided for viewing 
the workspace from which an image of the workspace xs 
obtained. Various other sensors and associated 

resoonders may be located at the worksite and operator.' s 



station, respectively, for transmission of pressure, 
tactile, heat, vibration and similar information for 
enhanced telepresence operation. 

Depending upon the application, different scaling 
5 may be" provided in the transmission of information 
between the operator's station and worksite. For 
example, for microassembly , microsurgery and like 
operations involving small part manipulation, optical 
and/or video magnification may be employed to provide an 
10 enlarged 3 -dimens ional image for viewing by the operator. 
With similar scaling between the hand operated means and 
manipulators, the perception of the operator is 
substantially that which a miniature operator would have 
were he at the worksite. 
15 rdt-F DESCRIPTION T v~ DRAWINGS 

The invention, together with other objects and 
advantages thereof, will be better understood from the 
following description considered with the accompanying 
drawings. It will be understood that the drawings are 
20 for purposes of illustration and example only, and that 
the invention is not limited thereto. In the drawings, 
wherein like reference characters refer to the same parts 
in the several views, 

Fig. i is a diagrammatic shewing of a telecperator 
25 svstem embodying the present invention including side 
elevationai views of a worksite and remote control 
operator's station; 

Fig. 2 is an enlarged rear elevationai view of the 
operator's station taken substantially along line 2-2 of 

3 0 Figure 1 ; 

Fig. 3 is an enlarged rear elevationai view of tne 
worksite taken substantially along line 3-3 of Figure 1; 

Fig. 4 is a simplified side elevationai view which 
is similar to Figure 1 and showing dimensional 
35 relationships between elements at the worksite and 
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e" 1 e-ents at the operator's station; 

Fig. 5 is a diagra-.n-.atic view £c illustrate visual 
perceotion by a miniature virtual eye, and Fig. 6 is a 
diagrammatic view to illustrate visual perception by the 
5 operator when image magnif ication is employed; 

Fig. 7 is a diagrammatic view which is similar to 
that of Fig- 1 but shewing the teleoperator system used 
for telepresence surgery; 

Fig. S is a rear elevational view of the operator's 

10 station shown in Fig- 7; 

Fig. 9 is a rear elevational view of the worksite 

shown in Fig . 7 ; 

Figs. 10 and 11 are fragmentary side elevational 
views of modified forms of operator's station and 
15 manipulator, respectively, having increased degrees of 
freedom ; 

Fig- 12 is a side elevational view of a modified 
form of operator's station wherein display means are 
positioned for direct viewing by the operator ; 
20 F ig. 13 is a rear elevational view of the modified 

form of operator's station shown in Fig. 12; and 

Fig. 14 shews a fragmentary portion of the insertion 
portion' of an endoscope for use with the present 

invention. . ^ ^ £ p-RE^t^^ E^^lVK^S 
Re^rence now is made to Figs. 1-3 wnerem .he 
teleoperator system is shown to include an operator's 
station 20 (Figs. 1 and 2) and worksite 22 (Figs. 1 and 
3). An coerator IS at the operator's station controls 
manipulator means 24 at the remote worksite. Manipulator 
means 24, comprising right and left manipulators 24R and 
24L, respectively, are used for manipulating objects, 
such as object 2 6 which is shown located on a platform, 
or base, 23 within a workspace 30 shown in broken lines. 
For purposes of illustration only, and not by way of 
limitation, the right manipulator 24R is shown to 
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comprise a housing 32R affixed to base 23 and from which 
housing a telescopic am 34R extends. The inner end 34R1 
of arm 34R is mounted for pivotal movement in any pivotal 
direction using conventional mounting means. rcr 
example, the inner end of arm 34R may be mounted for 
pivotal movement about a horizontal pivot axis 36 which 
pivot axis, in turn, is adapted for pivotal movement 
about vertical axis 3 S . 

Arm 34R includes telescopic inner section 34R1 and 
outer section 34R2, which outer section is adapted both 
for axial movement into and out of inner section 34R1 and 
for rotation about its longitudinal axis. An end 
effector 4 OR is carried at the cuter end of the arm 
which, for purposes of illustration, is shown to comprise 
15 a gripper.. Motor means, not shown, control pivotal 
"movement of arm 34R about pivot axes 36 and 33, axial and 
rotary movement of outer arm section 34R2 along and about 
the longitudinal axis of the arm, and opening and closing 
of gripper 40R. The motor means, together with motor 
20 control circuits for control of the motors, may be 
included in housing 32R. The motors are under control of 
a computer 4 2 connected thereto through right manipulator 
interface 44R and the above-mentioned motor control 
circuits . 

25 The left manipulator 24L is of substantially the 

same design as the right manipulator 24R and the same 
reference numerals, but with the suffix L instead of R, 
are used to identify similar parts. For purposes of 
illustration, the left end effector 40L, shown in Fig. 3, 
is seen to comprise cutting blades which operate to cut 
in the manner of a pair of scissor blades. 

The worksite is provided with a pair of video 
cameras 4 6R and 4 6L for viewing workspace 30 from 
different angles for production of stereoscopic signal 
therefrom at lines 43R and 4SL. The angle y 
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between the optical axes of the cameras shown in Fig. 3 
is substantially equal to the operator's interccular 
viewing angle y of an image of the workspace as shown in 
Fig. 2. 

5 The video camera outputs at lines 48R and 4SL are 

supplied to an image memory 50 for momentary storage of 
video fields of right and left images from the cameras. 
Fields of right and left images from image memory 50 are 
alternately supplied through left/right switch means 52 
10 to visual display means 54, such as a television monitor, 
for alternate display of the two images at the face 54A 
of the monitor. Timing and control means 56 provide 
timing and control signals to various elements of the 
system, including elements included in the stereographic 
display system, for signal timing and control of the 
system. If digital storage means 50 are employed, then 
conversion of the camera signal outputs to digital signal 
form by analog to digital converter means prior to 
storage, and conversion of the digital signal output from 
left/right switch means to analog signal form in 
preparation for display at monitor 54 may be employed. 

An electrooptical device 53 at the face of the 
display means 54 controls polarization of light received 
from display means 54 under control of a left/right 
25 synchronizing signal from timing and control unit 56. 
The left and right image fields are viewed by operator IS 
wearing a pair of passive polarized glasses 60 having 
right and left polarizing elements 62 and 64 polarized in 
orthogonal directions. The polarization of light from 
30 display 54 through electrooptical device 58 is 
synchronized field by field such that the right field is 
occluded from the left eye and the left field is occluded 
from the right eye for stereographic viewing by the 
operator. Other means for stereographic viewing of left 
35 and right image fields are well known, including, for 
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example, those using active stereographic glasses, which 
nay be used in the practice of this invention to provide 
the operator with a stereoscopic view of the remote 
workspace . 

5 The vertical deflection coil connections for monitor 

54 are reversed, causing the monitor- to scan from bottom 
to top thereby creating a top-to-bottom inverted image 
301 of workspace 30. Letters a, b, c and d are used to 
identify corresponding corners of the workspace 3 0 and 

10 inverted workspace image 301. The inverted workspace 
image 301 is viewed by the operator via a mirror 66 at 
the top of a table 68, which mirror inverts image 3 01 to 
return the image as viewed by the operator to an upright 
position. Looking downwardly in the direction of the 

15 mirror, the operator views a virtual image 30V of 
workspace 30. In accordance with one aspect of the 
present invention, the image viewed by the operator, 
which in the Fig. 1-3 embodiment comprises a virtual 
image, is located adjacent controller means 70 used by 

20 the operator for control of manipulator means 24 at the 
worksite . 

Controller means 70 are shewn located beneath the 
table top 63 and include right and left controllers 72R 
and 72L for control of the respective right and left 

25 manipulators 24R and 24L. The right and left controllers 
are of substantially the same design so that a 
description of one applies to both. As with the 
manipulators, the suffixes R and L are used to 
distinguish elements of the right controller from those 

30 of the left controller. For purposes of illustration, 
and not by way of limitation, the right controller 72R is 
shewn to comprise a housing 74R affixed to the bottom of 
table top 53 and from which hand-operated means 7 6R in 
the form of a telescopic control arm, or stick, extends. 

35 The right and left control arms 76R and 76L are 
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provided with the same degrees of freedom- as the 
associated manipulator arms 34R and 34L, respectively. 
For example, the inner end of control am 7 6R is mounted 
for pivotal movement about a horizontal pivot axis, 
5 corresponding to manipulator pivot axis 36, which axis, 
in turn, is adapted for pivotal movement about an 
intersecting vertical axis, corresponding to manipulator 
axis 33. Control arm 76R also includes inner section 
76R1 and outer section 76R2, which outer section is 

10 adapted both for axial movement into and out of inner 
section 76R1 and for rotation about its longitudinal 
axis. It will be apparent that the control arm 76R is 
provided with the same four degrees of freedom as the 
associated manipulator arm 34R. Additionally, sensor 

15 means 7SR are located adjacent the outer end of outer arm 
section 76R2 for use in controlling gripping action of 
gripper 40R. Similar sensor means 7SL adjacent the outer 
end of control arm'76L are adapted for use in controlling 
operation of scissor blades 40L. 

20 Right and left controllers 72R and 72L are included 

in a servomechanism system wherein mechanical motion of 
control arms 76R and 76L controls the position of 
manipulator arms 34R and 34L, and pressure on sensor 
means 73R and 7SL controls opening and closing of end 

25 effectors 40R and 40L, respectively. In Fig. 1, right 
and left hand controller, interfaces SOR and SOL, 
respectively, are shown for connection of the controllers 
to computer ; 42. Servcmechanisms for control of 

mechanical motion at a remote location are well known, 

30 including those which provide force and torque feedback 
from the manipulator to the hand-operated controller 
means. Any suitable prior art servomechanism may be used 
in the practice of the present invention, with those 
incorporating force and torque feedback being 

35 carticularly preferred for telepresence operation of the 
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system. In the illustrated system, right and left 
microphones are included at the worksite, outputs from 
which microphones are amplified by right and left 
amplifiers and supplied to right and left speakers at the 
5 operators' station for providing a stereophonic sound 
output to provide the operator with, an audio perspective 
present at the workspace. In Fig. 1, only the right 
channel of the stereophonic system is shown including 
right microphone S2R, right amplifier S6R and right 

10 soeaker S8R. The left microphone and speaker are located 
directly behind the respective right microphone and 
speaker at the worksite and operator's control station as 
viewed in Fig. 1. Obviously, earphones may be provided 
for use by the operator in place of the speakers which 

15 would help to block out external noises at the operator's 
control station. Also, in Fig. i a light shield 543 at 
the monitor is shown for blocking direct viewing of the 
monitor face by the operator. 

Reference now is made to Fig. 4 wherein a simplified 

20 diagrammatic view of the system illustrated in Figs. 1-3 
is shown and wherein various lengths and angular 
positions are identified by reference characters. In 
Fig. 4, the optical path length between the cameras and 
a point F at the workspace is identified by reference 

25 character L . A corresponding path length between the 
operator's eyes and point F at the virtual image of the 
workspace is identified by the distance a-b, where a is 
the distance from the eyes of the operator to mirror 66, 
and b is the distance from the mirror to point F at the 

30 virtual image. Other dimensions shown include the height 
G of the cameras above the pivot point of manipulator arm 
34R and corresponding height c of the operator's eyes 
above the pivot point of control arm 75R. With tne 
control arm 76R at length d, the manipulator arm 34R 

35 adjusts to length D. Similarly, with the control arm 76R 



ar an angle p A with the vertical, the manipulator arm 34R 
is positioned at the same angle from vertical. The angle 
from vertical at which the cameras view the workspace and 
the eyes view the virtual image of the workspace is 
5 identified by a . 

Between elements of the worksite and operator 
station, the following relationships pertain: 

a + b =kL, ( x ) 
d = kD , and ( 2 ) 

10 g = kG (3) 

where k is a scale factor constant. 
When k equals 1 such that a 4- b = L, d = D and g = G, no 
scaling of worksite dimensions is required. 

Any scale factor may be employed, the invention not 

15 being limited to full-scale manipulation. For example, 
the worksite can be small, including microscopic in size, 
in which case the optical parameters, including distance 
to object, interccular distance and focal length, and 
mechanical and dimensional parameters are appropriately 

20 scaled. 

By using appropriate scaling and image magnification 
and force and torque feedback, and by locating the image 
30V of the workspace 30 adjacent hand-operated control 
means 76R and 76L, the operator is provided with a strong 
25 sense of directly controlling the end effectors 40R and 
40L. The operator is provided with a sense that the end 
effectors 40R and 40L and respective control arms 76R and 
75L are substantially integral. This same sense of 
togetherness of the hand-operated control means and end 
effectors is not provided in prior art arrangements 
wherein the image viewed by the operator is not located 
adjacent the hand-operated control means. Even where the 
oricr art includes stereoscopic viewing and force and 
tcrcue feedback, there is a feeling of disconnectedness 
cf the hand motions from the visual image object being 
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worked upon. The present invention overcomes this sense 
of disconnectedness by locating the workspace image where 
the operator's hands appear to exercise direct control 
over the end effectors. 
5 For small-scale manipulation, such as required for 

surgical applications, it is desired to replicate the 
visual experience that a miniature observer would have 
were he closely adjacent the actual worksite. In Fig. 5, 
the virtual eye 90 of a hypothetical miniature observer 

10 is shown viewing an actual workspace. Light from a 
source at a point X, Y, Z in the actual workspace 
produces a stimulus on the miniature observer's eye 90 at 
a point identified as X'/K- In Fig. 6, an eye 92 of an 
actual operator is shown viewing an enlarged image of the 

15 virtual workspace produced by means of a video camera 94 
used to view the actual workspace. The illustrated 
camera includes a light-receiving lens 96 and solid state 
imaging device such as • a charge-coupled-device (CCD) 
array 98 where the point light source at X, Y, Z is shown 

20 imaged at point X L , Y L , Z i . With correct scaling, a 
corresponding light source is produced at point MX^, MY L , 
MZ; at either the real or apparent position of the face 
of the visual display which, due to stereoscopic 
operation of the system appears to the operator to 

25 originate from point MX, MY, MZ corresponding to point 
X,Y,Z at the actual workspace. At the retina of the 
actual eye 92, a stimulus is produced at point X' at 
proportionately the same position as point X'/ M at eve 90 
of the hypothetical observer. This relationship is 

30 ensured by selecting a correctly scaled camera distance 
and lens focal length such that the optical magnification 
.m = M/M v where M is the desired overall magnification 
and M v is the video magnification.' A typical video 
magnification, M v , which equals the ratio of the CCD- 

25 array 9S width to the display width, is about 40. 
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Reference new is made to Figs. 7 through 9 wherein 
a modified form of this invention is shown for medical 
use. Here, right and left manipulators 100R and 100L are 
shown which are under control of right and left 
5 controllers 102R and 102L, respectively. Elements of the 
imaging system are substantially -the same as those 
employed in the imaging system illustrated in Figs. 1-3 
described above except that an enlarged virtual image 
104V of actual workspace 104 is provided for viewing by 

10 the operator. Also, servomechanism elements for 

connection of the right and left controllers 102R and 
102L to the respective manipulators 100R and 100L are 
substantially the same as those described above with 
reference to Figs. 1-3. In the illustrated arrangement, 

15 the right and left manipulators are of substantially the 
same construction as are the right and left controllers, 
such that a description of one manipulator and one 
controller applies to both. Again, suffixes R and L are 
used to. distinguish between right and left elements 

20 thereof. 

The manipulators include outer control sections 
10 0 RA and 10 OLA and insertion sections 100R3 and 100L3, 
which insertion sections are adapted for insertion into 
a body cavity through cylindrical tubes, or cannulas, not 

25 shown. For purposes of illustration, the manipulators 
are shown inserted through the abdomen wall 106 of a 
subject. As is well understood, for laparoscopic 
surgical procedures, wall 106 is separated from internal 
organs by insufflation wherein a gas is introduced into 

3 0 the abdomen by any suitable means not shown. Manipulator 
motors and associated motor control circuits are 
contained in the cuter control sections 100RA and 100ILA 
of the manipulators for control of the insertion section. 
The manipulators, together with a laparoscope 108 for 

3 5 viewir.a organs within the cavity, are carried by a fixed 
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^ * c—r^ca 1 table uocn which the 
rail 110 forming par - c_ a s. — g_cci- 

subject is supported. 

The insertion sections .100R3 and 100L3 of the 
ra-oulators r.av be of substantially the sane design as 
manipulator arms 34R and 34L described above with 
le^ence to the Figs. 1-3 embodiment. The insertion 
sections are of relatively small siz-e for use inside the 
bodv rnsertion section 100R3 includes telescopic inner 
section' 112R1 and outer section 112*2, which outer 
section is adapted for both axial movement into and out 
or inner section 112R1 and for rotation about xts 
icncitudinal axis. End effectors 114R and 114L are 
carried at the outer ends of the respective ngnr. and 
le-t sections 112R2 and 112L2 for manipulation of organ 
116 The inner section 112R1 is adapted for pivotal 
movement about intersecting perpendicular axes located 
substantially at point P where the insertion section 
, r tersects wall 106. Exclusive of operation of ena 
e » ect0 r S H4R and 114L the manipulator arms each are 
p-ovided with four degrees of freedom, the same as in the 

„• ^ . effectors 114R and 

embodiment shown m rigs i-3 . 

1--4L simolv mav comprise, essentially, microsurgical 
irst-unents with their handles removed including, for 
evanole, retractors, electrcsurgical cutters and 
ccagulators, microforceps, microneedle holders, 
d-secting scissors, blades, irrigators, and sutures. 

Laparoscope 108 for viewing the workspace 104 is 
shown comprising an cuter operating section 108A and 

-ns^ T-e outer end section 12 0 of 
•> rse*"t icn section iOoJ. 

Irse-tVcn section 1083 is axialiy and rotatably movable 
w<th*n the inner end 12 2 thereof, and is provided with a 
pa w of imace transmission windows 124,124 for 

• „ w— -'--space 104. The laparoscope 
stereoscopic v.ew A1 .g Oi. « 

Provided with illuminating means, not shown for 
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illuminating the workspace, and with liquid inlet and 
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outl.* -=s, not .hovn. for flow of liquid past the 
„ <neK! . Video camera r.eans "i^in section 10.A a.e 
.:, 00 r..iv. to li=ht received through the viewing windows 
;„r" ceneration of left and right electronic ^ > - 
outout lines 4SR and 4^ for connection to ~ 
50." A ,agnified 3-dir.ensional i.age 104X is proouced a. 
disolav neans 54 for viewing by the operator wear ng 
cr o S s-polarized glasses 60 via nirror 66 As wi.h .he 
e*ocdir.ent shewn in Figs. 1-3. a virtual l.age 104V - 
the vcrKsoace 104 is produced adjacent control arr,s 1 OR 
and ,301. of controllers 102R and 102L. Control "OR 
and are of the sane type as control arr.s 76R ano 76. 

included in the rigs. 1-3 e==odir.ent descried above 
Thev include telescopic inner and outer sections 132R 
and"l32R, and 132L1 and 132L2. Sensor r.eans 134R and 
!34I. located adjacent the outer ends of the control arr.s 
control operation of end effectors 114R ano il«L. 
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manner described above with 
reference to Figs. 1-3. It here 



respectively, in the 



ill be noted that the 

= ^ich the irr.ace is viewed by the 
^mr-" 1 ^ *-cn vertical a<- ^nicn — - 

the object is viewed by the cameras. In the arrangement 
illustrated in Figs. 7-9, the operator is shown to view 
-he -age 104V at an angle e from vertical (Fig. 7) 
whereas the object 116 is shown as viewed directly 
downwardly. With no external reference, the sense o. 
ve^cal within a body is not particularly great, and no 
con-us ion is oroduced in the mind of the operator as a 
result of the different observer and camera viewing 
angles relative to vertical. 

With the Figs. 7-9 embodiment, not only is a 
nacn-«'i.d virtual image 104V of the workspace provided 

, _ v..*- rn-^-n 1 a^ms 13 OR and 
fo- viewing by the operator, c-- CCl,UiC - 

J ' _ ~ io-r--h the lencth of the manipulator 

130L of creamer 1 e . : c n *_.-ei.. - 
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insertion sections 100R3 and 100LB are employed. 
Servcmechanism scaling of axial movement of the 
telescopic control arms is provided such that axial 
extension or retraction thereof results in a smaller 
extension or retraction of the telescopic insertion 
sections. Angular pivotal motion of. the control arms 
130R and 130L produces the same angular pivotal motion of 
insertion sections. 100R3 and 100L3, and rotational 
movement of the end' sections 132R2 and 122L2 of the 
control arms produces the same rotational motion of end 
sections 112R2 and 112L2 of the insertion sections of the 
right and left manipulators, without scaling. This 
embodiment of the invention, with its magnified image, is 
of particular use in the area of microsurgery, and 
especially in those cases where the surgeon cannot reach 
an area- by hand because of size constraints. 

The present invention is not limited to use with 
manipulators having any particular number of degrees of 
freedom. Manipulators with different degrees of freedom 
which are well known in the art may be used in the 
practice of this invention. In Figs. 10 and 11, to which 
reference now is made a controller 140 and manipulator 
142, respectively, are shown which include a wrist joint 
to provide the same with additional freedom of movement. 
The illustrated controller 140 includes a housing 144 
affixed to the bottom of table top 6S upon which table 
mirror 66 is located. An enlarged virtual image 146V of 
actual workspace 146 is provided adjacent the operator's 
hand 14S viewable by the operator when looking downwardly 
onto the mirror 66 in a manner described above. 

A control arm I50L comprising inner and outer 
sections 150L1 and 150L2, respectively, is mounted within 
housing 144 for pivotal movement in any pivotal direction 
as indicated by intersecting double-headed arrows 152 and 
154. The outer section 150L2 is adapted for axial 
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noveaent into and out of inner section 150L1 in the 
direction of double-headed arrow 156. It also is adapted 
for rotation about its longitudinal axis in the direction 
of double-headed arrow 153. In this embodiment, the 
control arm includes an end section 160 pivotally 
attached to outer section 150L2 by wrist joint 162 for 
pivotal movement in the direction of double-headed arrow 
164. End section 160 comprises axially aligned inner and 
outer sections 160A and 160B, the outer section 160B of 
which is rotatable about its longitudinal axis in the 
direction of double-headed arrow 166. As with the above- 
described arrangements, sensor means 163 are located 
adjacent the free end of the control arm for operation of 
an end effector 170 at manipulator 142 shown in Fig. 11 : J 
Referring to Fig. 11, end effector 170 is shown to 
comprise a pair of movable jaws attached to a wrist 172 
comprising axially aligned links 172A and 1723. Outer 
link 172B is rotatable about its longitudinal axis 
relative to inner link 172A by motor means, not shown, in 
the direction of double-headed arrow 166M in response to 
rotation of section 160B of the hand-operated control 
unit in the direction of arrow 166. Wrist link 172A is 
pivotally attached ^manipulator forearm 1/4 ror pivotal 
^ movement in the direction of double-headed arrow 164M in 

25 response to pivotal movement of end section 160 of the 
hand-operated control means about pivot axis 162. 
Forearm 174 is longitudinally axially movable in the 
direction of double-headed arrow 156M in response to 
axial movement of outer section 150L2 of control arm 150L 
30 in the direction of double-headed arrow 156. It also is 
rotatable about its longitudinal axis in the direction of 
double-headed arrow 15SM in response to rotation of outer 
section 150L2 of control arm 150L in the direction of 
double-headed arrow 15S. Additionally, it is pivotally 
35 movable about point 176 in the directions of couble- 
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headed arrows 152M and 154M in response to pivotal 
movement of control arm 150L in the directions of double- 
headed arrows 152 and 154, respectively. For biomedical 
use, such as remote laparoscopic surgery, pivot point 176 
5 is substantially located at the level of abdominal wall 
178 through which the manipulator extends. In Fig. 11, 
manipulator arm 174 is shewn extending through a cannula 
180 which penetrates the abdominal wall. 

The outer operating end of the manipulator is 

10 adacted for attachment to a supporting rail, not shown, 
of the surgery table upon which the subject is supported. 
It includes an end effector drive motor 132 for opening 
and closing of gripper 170. Wrist drive motor 134 
controls oivotal movement of wrist 172 in the direction 

15 of double-headed arrow 164M, and extension drive motor 
136 controls axial movement of manipulator arm 174 in the 
direction of double-headed arrow 156M- Forearm pivotal 
control motors and linkages, identified generally by 
reference numeral 133, provide for pivotal movement of 

20 arm 174 about pivot point 176 in the directions of arrows 
152M and 1541-1. Pivotal motion about point 176 is 
provided by simultaneous lateral movement of the outer 
operating end of the manipulator and pivotal movement of 
arm 174. Movements are coordinated such that the center 

25 of rotation of forearm 174 is fixed in space at point 176 
at the level of the abdominal wall. 

Controller 140 and manipulator 142 are included in 
a system such as shewn in Figs. 7, 3 and 9 which includes 
a second controller and manipulator for use by the 

30 operator's right hand, and associated servomechanism 
means of any suitable type, net shewn, for remote control 
of the manipulators by the hand-operated controllers. 
Video camera means at the worksite, such as shown in Fig. 
9, together with display means, such as shown in Fig. 7, 

2 5 are emoloved for providing the operator with an image or 
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the workspace at a location adjacent the left and. right 
hand-operated control means. 5v^s_i.Dgji.an-ipulators with 
a wrist joint, an added degree cf freedom is provided for 
increased maneuverability and usefulness thereof . 
5 However, as noted above, the present invention is not 
limited to use with manipulators with any particular 
degree of freedom. 

Reference now is made to Figs. 12 and 13 wherein a 
modified form .of this invention is shown which provides 

10 for direct viewing ,of a 3 -dimensional image 2401 of a 
workspace, not shown. In Figs. 12 .and 13, only the 
operator's station is shown, which includes right and 
left controllers 242R and 242L and associated right and 
left hand-operated means 244R and 244L which may be of 

15 the same type as controllers and control arms described 
above. The operator's station is adapted for remote 
control of manipulators which also may be of the above- 
described type. The 3-dimensional image 2401 of the 
workspace is provided by visual display means 246 in 

20 conjunction with electrocptical device 53 at the face of 
the display means and cross-polarized glasses 60 worn by 
the operator, to which display means left and right video 
fields from left and right video cameras that view the 
workspace are alternately supplied, all in the manner 

25 described in detail above. End effector and object 
images 248 and 250, respectively, are shown within the 
workspace image as viewed by video cameras at the 
worksite. The display means 246 is located adjacent the 
left and right hand-operated means 244R and 244L for 

30 direct viewing by the operator. With this arrangement, 
the end effector and object images together with the 
hand-operated means 244R and 244L are simultaneously 
viewable by the operator. Since the hand-cperated means 
also are visible, the operator is provided with a visual 

35 sense of connection between the end effector means and 
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hand-operated means whereby they appear substantially as 

being integral. 

Reference new is made to Fig. 14 wherein the distal 
end portion, cr tip, 2 60 of the insertion section of an 
endoscope' is shown which is of substantially the sane 
type as shown in the above-mentioned publication entitled 
"Introduction to a New Project for National Research and 
Development Program (Large-Scale Project) in FY 1991" 
which endoscope may be used in the practice of the 
10 present invention. The insertion end of the endoscope 
includes a pair of spaced viewing windows 262R and 262L 
and an illumination source 264 for viewing and 
illuminating a workspace to be observed. Light received 
at v the windows is focused by objective lens means, not 
15 shewn, and transmitted through fiber-optic bundles to a 
pair of cameras at the operating end of the endoscope, 
not shown. The camera outputs are converted to a 3- 
dimensional image of the workspace which image is located 
adjacent hand-operated means at the operator's station, 
20 not shown. Right and left steerable catheters 26SR and 
2 6SL pass through accessory channels in the endoscope 
body, which catheters are adapted for extension from the 
i distal end portion, as illustrated. End effectors 270R 
and 270L are provided at the ends of the catheters which 
25 . mav comprise conventional endoscopic instruments. Force 
sensors, not shown, also are inserted through the 
endoscope channels. Steerable catheters which include 
control wires for controlling bending of the catheters 
and operation of an end effector suitable for use with 
3 0 this invention are well known. Control motors for 

f the control wires are provided at the 
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operating end of the endoscope, which meters are included 
in a serve-ecr.anism of a type described above for 
operation of the steerable catheters and associated end 
effectors from a remote operator's station. As with the 
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other embodiments, the interfacing computer "in the 
servomechanism system remaps the operator's hand motion 
into the coordinate system of the end effectors, and 
images of the end effectors are viewable adjacent the 
5 hand-operated controllers in a manner described above. 
With this embodiment, the operator has the sensation of 
reaching through the endoscope to put his hands directly 
on the end effectors for control thereof. Endoscopes of 
different types may be employed in this embodiment of the 

10 invention so long as they include one or more accessory 
channels for use in control of end effector means, and 
suitable viewing means for use in providing a visual 
display of the workspace. For example, gastric, 

colonscopic, and like type , endoscopes may be employed. 

^5 The invention having been described in detail in 

accordance with requirements of the Patent Statutes, 
various other changes and modifications will suggest 
themselves to those skilled in this art. For example, as 
noted above, the invention may include the use of tactile 

20 feedback to provide the subtle sensations for palpation 
and for manipulating tissues and instruments. To provide 
this feedback, tactile sensor arrays may be included on 
the end effectors which are coupled to tactile sensor 
stimulator arrays on the hand-operated control means, 

25 which reproduce the tactile 'sensation on the operator's 
hands. A variety of transduction technologies for 
telecperator tactile sensing are known including 
resistive/ conductive , semiconductor , piezoelectric 
caoacitive and photoelectric. Hand-operated control 

30 means and manipulators of different types may be employed 
using a wide variety of well-known mechanisms and 
electromechanical elements including, for example, 
gimbals, linkages, pulleys, cables, drive belts and 
bands, gears, optical or electromagnetic position 

3 5 encoders, and angular and linear motors. Force feedback 
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to the ooerator requires use of body contact with hand- 
operated control neans. Both hand grip type hand 
controllers such as those illustrated, and control brace 
type hand controllers are veil adapted for use with the 

5 present invention for force feedback to the operator. 
Control brace hand controllers include use of structures 
with positive sensors mounted on the operator at joints 
for measuring joint angles. Force feedback then can be 
applied to each joint. Similarly, light fabric gloves 

10 with variable-resistance or fiber-optic flex sensors 
mounted on the joints for measuring bending of individual 
fingers may be used. Gloves of this type also may be 
provided with force feedback to provide for telepresence 
"interaction with real objects. Regardless of the type of 

15 hand-operated control means employed, an image of the 
workspace is produced adjacent thereto to provide the 
operator with a sense that the end effector means and 
hand-operated control means are substantially integral. 
Also, as noted above, servomechanisms of many different 

20 types are well known in the robotic and teleoperator 
system arts, and the invention is not limited to any 
particular type. Those that include force and torque, 
feedback to the operator are preferred to contribute to 
a telepresence sense of operation. In addition, many 

25 different means for producing a stereoscopic image of the 
workspace are known. For example, instead of using two 
cameras, a single camera may be employed together with 
switched cress-polarizing elements in the image receiving 
path. In this case, a pair of spaced stereoscopic lenses 

30 are used for viewing the workspace from different angles 
and providing first and second images thereof to the 
camera. In the Fig. 9 arrangement, wherein a laparoscope 
is shown, ether types cf endoscopes may be used for 
viewing the workspace. As noted above, the invention is 

35 not limited to any particular application or use. In the 
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biomedical field, uses include, for example, open 
surgery, including surgery from a remote location, 
microsurgery, and minimum invasive surgery such as 
laparoscopic and endoscopic surgery. Laboratory use 
including microscopic manipulation also is contemplated, 
industrial use of the invention include, for example, 
hazardous materials handling, remote operations, 
microassembly, and the like. Military and undersea use 
of the teleoperator system of this system are apparent. 
It is intended that the above and other such changes and 
modifications shall fall within the spirit and scope of 
the invention defined in the appended claims. 



